Điều kiện tối ưu và đối ngẫu trong tối ưu không trơn và các vấn đề liên quan

VIETNAM NATIONAL UNIVERSITY - HO CHI MINH CITY UNIVERSITY OF SCIENCE NGUYEN MINH TUNG Optimality conditions and duality in nonsmooth optimization and related problems PhD THESIS IN MATHEMATICS Ho Chi Minh City - 2015 VIETNAM NATIONAL UNIVERSITY HO CHI MINH CITY UNIVERSITY OF SCIENCE NGUYEN MINH TUNG Optimality conditions and duality in nonsmooth optimization and related problems Major: Optimization Theory Code: 62 46 20 01 First examiner: Associate Prof. LAM QUOC ANH Second examiner: Dr. NGUYEN BA THI Third examiner: Associate Prof. NGUYEN NGOC HAI First independent examiner: Associate Prof.

DO VAN LUU Second independent examiner: Dr. LE THANH TUNG SCIENTIFIC SUPERVISOR: Prof. PHAN QUOC KHANH Ho Chi Minh City - 2015 Acknowledgements The completion of this doctoral dissertation would not have been possible without the support of several people. I would like to express my sincere gratitude to all of them.

First, I want to express my deepest gratitude to my supervisor, Professor Phan Quoc Khanh, for his valuable guidance, scholarly inputs, and consistent encouragement I received throughout the research work. From deciding on the research topic in the beginning to the process of actual writing of the thesis, he offered his unreserved help and guided through every step of my work. He provided me with inspiring and insightful guidance, without which this study would never have reached a successful completion. Second, I am very pleased to extend my thanks to the reviewers of this the- sis.

Their comments, remarks and questions have truly improved the quality of this manuscript. I would like to thank the professors who agreed to be on the jury judging my thesis defence. To my colleagues, I would like to express my thankfulness to Dr. Nguyen Dinh Tuan who extended his support in a very spe- cial way, and I benefited a lot from his personal and scholarly interactions, his suggestions at various points of my research program.

My thanks also go to the Faculty of Mathematics and Computer Science at the University of Science, Viet- nam National University Ho Chi Minh City for their support during the period of preparation of my thesis. Finally, I owe a lot to my parents, my brother, my older sister and my younger sister who support, encourage and help me at every stage of my personal and academic life, and long to see this achievement come true. They always make sure I am provided with a carefree environment where I can devote myself entirely to my study. I am really lucky to be a member of such a fantastic family.

i Contents Acknowledgements i List of Symbols and Notations iv Preface vi 1 Premilinaries 1 1.1 Some definitions in set-valued analysis .2 First-order and second-order tangent sets .3 Set-valued optimization problems. 5 2 Second-order optimality conditions with the envelope-like effect in set-valued optimization 8 2.2 Approximations as generalized derivatives .3 Necessary optimality conditions .4 Sufficient optimality conditions. 19 3 Second-order KKT optimality conditions with the envelope-like effect in some general set-valued optimization problems 30 3.2 Set-valued optimization problems with generalized inequality con- straints .1 Second-order necessary optimality conditions .2 Second-order sufficiency optimality conditions .3 Set-valued optimization problems with mixed constraints .1 Second-order necessary optimality conditions .2 Second-order sufficiency optimality conditions. 63 ii 4 Second-order asymptotic contingent epiderivatives and set-valued optimization problems 66 4.2 Second-order asymptotic contingent epiderivatives.

79 5 Optimality conditions and duality for nonsmooth vector equilib- rium problems with constraints 82 5.2 Mond-Weir duality .1 Constrained vector minimization .2 Cone saddle point problems. 103 Conclusions 105 Expected further works 107 List of the author’s publications related to the thesis 108 List of the author’s conference reports related to the thesis 109 References 109 iii List of Symbols and Notations N the natural numbers Q the rational numbers R the real numbers R = R ∪ {−∞, +∞} the extended real numbers Rn the n-dimensional Banach space Rn+ the non-negative orthant of Rn k · kX the norm of a space X dimX the dimension of a space X BX the closed unit ball of a space X BX (x0 , r) the closed ball with the center x0 and the radius r of a space X X∗ the dual space of a space X L(X, Y ) the space of continuous linear mappings from X to Y B(X, X, Y ) the space of continuous bilinear mappings from X × X to Y C∗ the dual cone of C C ∗i the quasi-interior dual cone of C N (D, z0 ) the normal cone of D at z0 {xn } the sequence with elements xn xn → x xn converges to x clM the closure of a set M intM the interior of a set M coreM the algebraic interior of a set M bdM the boundary of a set M convM the convex hull of a set M coneM the cone generated by M T (M, x0 ) the contingent cone of M at x0 IT (M, x0 ) the interior tangent cone of M at x0 T 2 (M, x0 , u) the second-order contingent set of M at x0 in direction u A2 (M, x0 , u) the second-order adjacent set of M at x0 in direction u IT 2 (M, x0 , u) the second-order interior set of M at x0 in direction u 00 T (M, x0 , u) the second-order asymptotic contingent set of M at x0 in direction u 00 A (M, x0 , u) the second-order asymptotic adjacent set of M at x0 in direction u 00 IT (M, x0 , u) the second-order asymptotic interior set of M at x0 in direction u f :X→Y a single-valued mapping from X to Y F :X⇒Y a set-valued mapping from X to Y F −1 : Y ⇒ X the inverse mapping of F : X ⇒ Y domF the domain of F gphF the graph of F iv epiF the epigraph of F F+ the profile/epigraphic map of F f 0 (x0 ) the first-order Fréchet derivative of f at x0 f 00 (x0 ) the second-order Fréchet derivative of f at x0 df (x0 , u) the Hadamard directional derivative of f at x0 d2 f (x0 , u)(x) the second-order Hadamard directional derivative of f at x0 in direction (u; x) 00 d f (x0 , u)(x) the asymptotic second-order Hadamard directional derivative of f at x0 in direction (u; x) DF (x0 , y0 ) the contingent derivative of F at (x0 , y0 ) Dl F (x0 , y0 ) the lower contingent derivative of F at (x0 , y0 ) D2 F (x0 , y0 , u, v) the second-order contingent derivative of F at (x0 ; y0 ) in direction (u; v) D[(2) F (x0 , y0 , u, v) the second-order adjacent derivative of F at (x0 ; y0 ) in direction (u; v) Dl2 F (x0 , y0 , u, v) the second-order lower derivative of F at (x0 ; y0 ) in direction (u; v) 00 D F (x0 , y0 , u, v) the asymptotic second-order contingent derivative of F at (x0 ; y0 ) in direction (u; v) 00 D[( ) F (x0 , y0 , u, v) the asymptotic second-order adjacent derivative of F at (x0 ; y0 ) in direction (u; v) 00 Dl F (x0 , y0 , u, v) the asymptotic second-order lower derivative of F at (x0 ; y0 ) in direction (u; v) EDF (x0 , y0 ) the contingent epiderivative of F at (x0 , y0 ) ED2 F (x0 , y0 , u, v) the second-order contingent epiderivative of F at (x0 ; y0 ) in direction (u; v) EDc(2) F (x0 , y0 , u, v) the composed second-order contingent epiderivative of F at (x0 ; y0 ) in direction (u; v) 00 ED F (x0 , y0 , u, v) the asymptotic second-order contingent epiderivative of F at (x0 ; y0 ) in direction (u; v) ≤C the order relation generated by C d(x, M ) the distance from x to M u⊥ the orthogonal complement of u l. upper semicontinuous t↓0 t > 0 and t → 0 |·| the absolute value h·, ·i the canonical pairing of any pair of dual spaces [·, ·] the close interval (·, ·) the open interval (·, ·] and [·, ·) the half-open intervals ∅ the empty set ∀ for all ∃ there exists 2 the end of a proof v Preface In the intensively developed optimization theory and applications, set-valued vec- tor problems have been taking a remarkably and continuously increasing part in the enormous number of contributions to the literature. Vector optimization, including multiobjective problems (as usual the terminology “multiobjective” is used when the number of objectives is finite, and “vector” for the general case), began to be studied more than a century ago, in mathematical applications in economics first, while set-valued optimization began to attract attention from researchers about five decades ago.

This thesis is devoted to set-valued vector op- timization. On the other hand, most practical problems are nonsmooth, i., the involved maps and functions are not Fréchet and Gateaux differentiable. Hence, in our work we always consider nonsmooth problems, like most important con- tributions to the literature in recent years. Besides derivative-free investigations, the main approach is introducing generalized derivatives to replace the classical ones, and developing their applications.

There have been numerous notions of generalized derivatives, each of which is valuable in some classes of problems but none is universal, due to the diversity of practical situations. Until now, many derivative-like notions have been proposed and applied to investigate optimality conditions in nonsmooth problems (see [7, 14, 15, 31, 38, 41, 79, 80] for first-order conditions, [4, 43, 56, 57, 72, 73, 95, 101] for higher-order conditions, and the references therein). In the pathbreaking paper [14], Corley employed the contingent and circatangent derivatives to establish a first-order Fritz John necessary optimality condition. Jahn and Rauh [41] introduced the notion of contingent epiderivative, which is an extension of directional derivative, but it is a single map and its existence is still an difficult problem.

For overcoming this difficulty, Chen and Jahn [15] proposed a generalized contingent epiderivative of set-valued maps, and it was applied to obtain first-order optimality conditions in [15, 41]. By using contingent epiderivatives, Gotz and Jahn in [31] obtained a first-order Karush-Kuhn-Tucker (KKT) necessary optimality condition. With the well-known Dubovitski-Milutin approach, Issac and Khan in [38] established a Lagrange multiplier rule for set-valued optimization with generalized inequality constraints. Another fruitful approach in set-valued optimization is the dual space approach initiated by Mordukhovich (see [79, 80]).

In our contributions we apply several kinds of generalized derivatives as mathematical tools in order to best serve our purpose. Depending on particular research situations, we even propose new concepts of generalized derivatives to suit our problems under consideration. Recently, second-order optimality conditions for scalar and vector optimiza- tion problems have been intensively developed (for example, we cite [2, 10, 13, vi 26, 34, 50, 53, 54, 61, 62, 75, 82, 83, 91, 93, 100] and the references therein) be- cause they refine the first-order by second-order information which is very helpful for recognizing optimal solutions as well as for designing numerical algorithms for computing them. We observe, in most related contributions in the literature, that the core of second-order necessary optimality conditions is a direct extension of the classical result in calculus that the second derivative of the objective map (or the Lagrange map in constrained problems) at minimizers is nonnegative.

Kawasaki in [50] discovered that the second derivative of the Lagrangian at the minimal solution may be strictly negative in certain critical directions. He called this phenomenon the envelope-like effect. The Kawasaki result was developed in [13, 46, 83] for C2 scalar programming, in [34, 61] for nonsmooth multiobjective programming, and in [62, 93] for infinite-dimensional nonsmooth optimization.